test(e2e): pin prompt-cache, service-tier, and cost-header billing as permanent regressions - #37607

TLDR

Problem this solves:

• Six shipped billing fixes had zero e2e coverage
• Cache-read, cache-write, and tier rates could silently regress
• Per-component cost headers and breakdown rows went unasserted

How it solves it:

• Six live tests, one per shipped fix, real OpenAI caching
• Each deployment gets distinct per-component rates, so wrong-rate bills cannot pass
• Assert the row's gross input and the header's net components separately

User Flow

Before: back when these regressions were live, a customer running a long cached prompt through the gateway read back a bill that was wrong in several places at once, and nothing in the suite would have caught any of them coming back

1. They send POST https://litellm-domain/v1/chat/completions with a ~3.6k-token prefix and see "cache_write_tokens": 3627 come back under usage.prompt_tokens_details
2. They read GET https://litellm-domain/spend/logs?request_id={id} and the whole prompt is billed at the plain input rate, with nothing on the row separating what the cache write cost
3. They send the same prefix again, get "cached_tokens": 2816 back, and the row still shows no cache-read cost, so the discounted tokens bill at full input price
4. They repeat that second call with "stream": true and the usage comes back with the cached-token count dropped, so a streamed cache hit costs the same as a cold one
5. They send the same prefix to POST https://litellm-domain/v1/messages against a Responses-only model; the reply says "cache_read_input_tokens": 7936 but the spend row records zero cached tokens
6. They add "service_tier": "priority", the reply confirms "service_tier": "priority", and the reasoning tokens inside the answer are still billed at the standard output rate
7. They read the x-litellm-response-cost-* headers off any of those replies and the per-component values do not add up to x-litellm-response-cost

After: every one of those readings is right today, and this PR is what holds them there, since each step above is now asserted by a live test that goes red if the number moves

1. The same POST https://litellm-domain/v1/chat/completions returns "cache_write_tokens": 3627
2. GET https://litellm-domain/spend/logs?request_id={id} shows the cache-write tokens priced at the cache-creation rate, listed separately from the fresh input tokens
3. The repeat call's row prices the 2816 cached tokens at the cache-read rate and the rest at the input rate
4. The streamed repeat keeps its cached-token count, so it bills at the same discount as the non-streamed one
5. The POST https://litellm-domain/v1/messages call's "cache_read_input_tokens": 7936 shows up on the spend row and is billed at the cache-read rate
6. The priority call bills input, output, and the reasoning tokens inside output at the priority rates, and the row records "service_tier": "priority"
7. The x-litellm-response-cost-* component headers add up exactly to x-litellm-response-cost

Relevant issues

Pins the fixes for #34046, #31686, #34812, #34957, #35923, #35925, and #36965

Linear ticket

Resolves LIT-5869

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

This PR adds tests only, so no product code differs between the two sides. Before is the merge base showing the behaviors have no coverage; After is the live run at the tip proving the numbers those tests assert are the numbers a real proxy produces

Shared setup: a proxy from this branch on http://localhost:48206 against a local Postgres, and four deployments registered at runtime, each with rates ~100x canonical and distinct per component so a bill computed from the wrong rate cannot match: input 4e-05, output 8e-05, cache-read 1e-05, cache-creation 5e-05, and on the tier deployment priority input 6e-05 and priority output 1.6e-04. All calls use a virtual key from POST /key/generate, and all traffic hits real OpenAI. The two legs that assert on reasoning cost send "reasoning_effort": "high", the same as the tests do

Before (0b37454)

1. uv run pytest tests/e2e/quota_management/spend_tracking --collect-only -q | tail -3 -> the suite collects nothing for prompt-cache billing, service-tier pricing, cost headers, or passthrough stream cost
2. python -m coverage_registry.collector -> the six registry rows added here are absent, so none of these behaviors counts as covered and a regression in any of them ships with a green suite

After (975a680)

Case 1: cache-write tokens bill at the cache-creation rate (#34046)

1. curl -s $P/chat/completions -H "Authorization: Bearer $KEY" -d @qa3_write.json -> id chatcmpl-EEt11sR342HaQnD4X7rd2g7a9AKCA, "prompt_tokens": 3630, "cache_write_tokens": 3627
2. curl -s "$P/spend/logs?request_id=chatcmpl-EEt11sR342HaQnD4X7rd2g7a9AKCA" -H "Authorization: Bearer sk-1234" -> "cache_creation_cost": 0.18135 (3627 x 5e-05), "input_cost": 0.18147, so the 3 fresh tokens are the remaining 0.00012 at 4e-05, and "total_cost": 0.18395 equals input plus the 0.00248 output

Case 2: the breakdown itemizes cache-read and reasoning (#31686)

1. curl -s $P/chat/completions ... -d @qa10_read_prime.json then -d @qa10_read_measure.json, both bodies carrying "reasoning_effort": "high" -> the measured call returns "cached_tokens": 2816, "reasoning_tokens": 67
2. curl -s "$P/spend/logs?request_id=chatcmpl-EEuyXBfTmaempDNVgnKQ16C4nB7TU" ... -> "cache_read_cost": 0.02816 (2816 x 1e-05), "reasoning_cost": 0.00536 (67 x 8e-05) sitting inside "output_cost": 0.00624, and "input_cost": 0.0608 less the cache-read cost leaves the 816 fresh tokens at 4e-05

Priming at a different reasoning_effort than the measured call produces no cache read at all, so both bodies carry the same value. That is why the test sends it on its prime call too

Case 3: a streamed cache read keeps its discount (#34812)

1. curl -s $P/chat/completions -H "Authorization: Bearer $KEY" -d @qa9_prime.json primes a fresh 1200-word prefix -> cached_tokens 0
2. curl -sN $P/chat/completions -H "Authorization: Bearer $KEY" -d @qa9_stream.json sends the same prefix with "stream": true and no stream_options -> first chunk id chatcmpl-EEtJmzT2FmyV0Db1tP6Bh2jj2GVvs
3. curl -s "$P/spend/logs?request_id=chatcmpl-EEtJmzT2FmyV0Db1tP6Bh2jj2GVvs" -H "Authorization: Bearer sk-1234" -> "cache_read_input_tokens": 2816 survived the stream, "cache_read_cost": 0.02816 (2816 x 1e-05), and the 815 fresh tokens are 0.0326 at 4e-05, so "input_cost": 0.06076 is exactly those two added together and "total_cost": 0.06212 is that plus the 0.00136 output. At full input price this call would have billed 0.14524, so the discount is worth 2.3x here

OpenAI's implicit caching is best-effort, so this leg was driven with the same retry-with-a-fresh-prefix loop the test uses; it hit on the first attempt above, and a separate control run put a plain stream and an include_usage stream on one warm prefix to confirm the discount does not depend on the client asking for usage

Case 4: cache tokens survive the /v1/messages bridge onto a Responses-only model (#34957)

1. curl -s $P/v1/messages -H "Authorization: Bearer $KEY" -d @qa3_bridge_prime.json -> "input_tokens": 9030
2. curl -s $P/v1/messages -H "Authorization: Bearer $KEY" -d @qa3_bridge_read.json -> "input_tokens": 1095, "cache_read_input_tokens": 7936
3. curl -s "$P/spend/logs?api_key=$KEY" ... -> the bridged row carries cache_read 7936 with "cache_read_cost": 0.07936 and an "input_cost" whose remainder after the cache read is the fresh tokens at 4e-05

Case 5: priority tier bills every component at the priority rate, reasoning included (#35923, #35925)

1. curl -s $P/chat/completions -H "Authorization: Bearer $KEY" -d @qa10_tier.json -> "service_tier": "priority", "prompt_tokens": 34, "completion_tokens": 51, "reasoning_tokens": 40
2. curl -s "$P/spend/logs?request_id=chatcmpl-EEuyiqkNSoFQX30mN5FdHZQLBXlcb" ... -> "service_tier": "priority" on the row, "input_cost": 0.00204 (34 x 6e-05), "output_cost": 0.00816 (51 x 1.6e-04), and "reasoning_cost": 0.0064 (40 x 1.6e-04). The standard rates would have billed 0.00136 and 0.00408, so none of the three landed on the default tier

Case 6: the per-component cost headers add up to the total (#36965)

1. curl -s -D- $P/chat/completions -H "Authorization: Bearer $KEY" -d @qa10_read_measure.json -> x-litellm-response-cost: 0.06704, -input: 0.03264, -output: 0.00624, -cache-read: 0.02816, -tool-usage: 0.0
2. 0.03264 + 0.00624 + 0.02816 + 0.0 = 0.06704, equal to the total header. Note the header's input component is net of cache while the spend row's input_cost is gross (0.0608 on the same call), which is why the two surfaces are asserted differently

All six tests also ran green against this same proxy at this commit: 6 passed in 139.10s

Type

✅ Test

Caveats (if any)

• Two sibling behaviors are not e2e reachable and stay unit-covered, see below
• OpenAI caching is best-effort, so cache tests retry with a fresh prefix

Two behaviors from the same batch were investigated and deliberately left out, both because no live proxy can reach them:

• The scientific-notation cost coercion (fix(router): cast model_info cost values to float in _set_model_group_info #33556) cannot be triggered through any public route. POST /model/new and POST /model/update bind the body to a model whose cost fields are typed float | None, so pydantic coerces "3e-06" at the HTTP boundary, and the boot-config path is coerced too before it is ever read. The PR's own unit tests have to patch the reader to inject a string, which the e2e harness forbids
• The blocked-response usage on /v1/responses (fix: report real token usage on guardrail-blocked /v1/responses replies #36907) only differs from the old behavior on a post-call block that carries the upstream response. Every guardrail available in-tree either answers 400 outright or blocks pre-call with no upstream response, which yields zeros both before and after the fix. Reaching the fixed branch needs Bedrock with disable_exception_on_block, Straiker, or Rubrik, all external and credentialed

A seventh registry row, quota_management.spend_tracking.passthrough_stream.injects_usage_cost, ships uncovered on purpose. The behavior it names (#36503) only shows up with litellm_settings.include_cost_in_streaming_usage turned on, and that is a boot-time setting the shared e2e stack does not enable today, so a test written against that stack now cannot reach the branch it is meant to pin. That is observed rather than assumed: an earlier commit on this branch did carry such a test, and it came back from CI with cost=None on the final usage frame, which is exactly the pre-fix reading. A change to the stack's configuration is open and has to land first. The row stays in the registry so the collector keeps reporting it as a gap instead of letting the behavior quietly disappear from the denominator

One doc nit found on the way, not changed here: the input_cost field comment in litellm/types/utils.py reads "Cost of raw (non-cached) input tokens only", which is true of the response header but not of the spend row, where the same name is gross of cache

QA runbook

• tests/e2e/quota_management/spend_tracking/test_cache_cost_accounting_e2e.py::TestCacheCostAccounting::test_cache_write_tokens_billed_at_cache_creation_rate - a fresh ~3.6k-token prompt's cache-write tokens bill at the cache-creation rate, not the input rate

  • POST /model/new with openai/gpt-5.6-luna and input_cost_per_token 4e-05, cache_creation_input_token_cost 5e-05
  • POST /v1/chat/completions with a ~1200-word unique prefix and expect the spend row to carry cache-creation tokens above zero
  • GET /spend/logs?request_id={id} and expect cache_creation_cost to equal those tokens times 5e-05, with input_cost minus that leaving the fresh tokens at 4e-05
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/quota_management/spend_tracking/test_cache_cost_accounting_e2e.py::TestCacheCostAccounting::test_cost_breakdown_reports_component_costs - the row itemizes cache-read and reasoning cost, with reasoning a subset of output

  • POST /model/new with openai/gpt-5.5 and the four distinct rates
  • POST /v1/chat/completions twice with the same prefix, a different trailing question each time, and expect the second call's row to carry cached tokens
  • GET /spend/logs?request_id={id} and expect cache_read_cost at 1e-05 per cached token, reasoning_cost at 8e-05 per reasoning token, and reasoning_cost no greater than output_cost
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/quota_management/spend_tracking/test_cache_cost_accounting_e2e.py::TestCacheCostAccounting::test_streaming_cache_read_billed_at_cache_read_rate - a streamed cache hit gets the same discount as a non-streamed one

  • Prime the prefix with a non-streamed call, then send the same prefix with "stream": true
  • Take the id off the first SSE chunk and GET /spend/logs?request_id={that id}
  • Expect a non-zero cached-token count on the row priced at 1e-05, not at the 4e-05 input rate
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/quota_management/spend_tracking/test_cache_cost_accounting_e2e.py::TestCacheCostAccounting::test_messages_bridge_keeps_cache_tokens - cache tokens survive the anthropic-messages to Responses bridge and keep their discount

  • POST /model/new with openai/gpt-5.3-codex, which is Responses-only and so forces the bridge
  • POST /v1/messages twice with a ~3000-word shared prefix and expect cache_read_input_tokens above zero on the second
  • GET /spend/logs?api_key={key} and expect a row for the key carrying cache-read tokens with cache_read_cost at 1e-05 each (the bill is filed under a different id than the response's, so query by key)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/quota_management/spend_tracking/test_service_tier_pricing_e2e.py::TestServiceTierPricing::test_priority_tier_bills_priority_rates - a priority call bills input, output, and reasoning at the priority rates

  • POST /model/new with both standard rates and input_cost_per_token_priority 6e-05, output_cost_per_token_priority 1.6e-04
  • POST /v1/chat/completions with "service_tier": "priority" and a reasoning question, and expect the response to echo "service_tier": "priority"
  • GET /spend/logs?request_id={id} and expect service_tier priority on the row, output and reasoning at 1.6e-04, input at 6e-05
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/quota_management/spend_tracking/test_cost_headers_e2e.py::TestCostHeaders::test_component_cost_headers_sum_to_total - the per-component cost headers add up to the total header

  • POST /model/new with openai/gpt-5.5 and the same four distinct rates
  • POST /v1/chat/completions twice with a shared ~1200-word prefix, priming first, and pass -D- so the measured call's headers are visible (the test reads them off the raw transport, which the typed client drops)
  • Read x-litellm-response-cost and the -input, -cache-read, -cache-creation, -output, -tool-usage components
  • Expect the five components to sum to the total, the cache-read header at 1e-05 per cached token, and the input header covering only the fresh tokens at 4e-05 (these are net of cache, unlike the spend row's gross input_cost)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

CI note, so the red gate is not mistaken for this diff: every e2e run at this tip came back 1 failed, 640 passed, 58 skipped, and the one failure is llm_translation/test_audio_transcriptions_e2e.py::TestAudioTranscriptions::test_missing_model_returns_error, which fails the same way on every PR that runs the suite today. All six tests added here passed against the shared stack. #37565 carries the one-line assertion fix for that test, so this gate goes green once it merges. Separately, buildkite/e2e-tests is not one of litellm_internal_staging's required checks, so the red X does not block merging

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

CHECKED: live-pr-risk at 975a680. Five of the six files are new or registry-only, and the one shared file is tests/e2e/models.py, where every change is additive: AnthropicMessagesResponse.usage plus four optional cost fields on LiteLLMParamsBody, all defaulting to None, with no field renamed, retyped, or given a new default. Request bodies serialize with exclude_none=True, so no existing suite's call changes on the wire, and no suite outside this PR reads .usage off a /v1/messages response. The full e2e suite collects clean on the head (695 of 697, two deselected, 0 errors), the coverage collector runs --strict without a failure, basedpyright over tests/e2e reports 0 errors, and all six cases were driven live against a real proxy, database, and OpenAI

• 975a680 passes flake test (ran 4 times). All six tests added here passed in every run, and each build's only failure is the pre-existing audio-transcriptions assertion that feat(e2e): move record/replay to the provider edge (LIT-5745) #37565 fixes